Clinical Uses of Cuffless Blood Pressure Monitoring Devices Using Photoplethysmography as a Promising Ambulatory Telemonitoring Tool: A Systematic Review and Meta-Analysis

Introduction: As check-ups in healthcare facilities are much arduous during the pandemic including blood pressure (BP) control, an alternative is urgently needed to replace the use of disturbing cuff-based office and ambulatory BP monitoring (BPM) devices. With the advancement of telemedicine, real-time checking and reporting of blood pressure may be potentially achieved using photoplethysmography (PPG) technology in cuffless devices. Therefore, this study evaluated the accuracy of these devices compared to the cuff-based BPM devices. Method(s): This systematic review and meta-analysis was conducted based on the PRISMA 2020 guideline through multiple databases using Rayyan according to the prearranged inclusion and exclusion criteria, yielding six clinical studies to be included in the final review and analysis. Result(s): Overall fixed-effect meta-analysis of all studies (total of 319 subjects) presented small differences between cuffless and cuff-based devices, showing promising accuracy according to the current medical instrumentation guideline both in measuring systolic BP (SMD: 0.23 mmHg [95% CI: 0.07-0.39], p = 0.004;I2= 0%, p = 0.55) and diastolic BP (SMD: 0.27 mmHg [95% CI: 0.11-0.43], p = 0.0007;I2= 39%, p = 0.14). Discussion(s): PPG itself is a noninvasive technology, consisting of an infrared-emitting light source and a photodetector to measure the blood-reflected light intensity. Despite its ease in equipment, it measures BP accurately without being influenced by various positions and activities. Moreover, the data can be accessed real-time by both users and healthcare providers. Conclusion(s): In summary, cuffless PPG BPM devices have the potential in becoming a telemonitoring device for ambulatory patients for its accuracy. Its presence may be the answer to current restriction towards healthcare access during the COVID-19 pandemic. Therefore, in order to further confirm our findings, more clinical studies with various settings are encouraged to be held.

The Pi-CON Methodology Applied: Operator Errors and Preference Tracking of a Novel Ubiquitous Vital Signs Sensor and Its User Interface

Remote Patient Monitoring has enjoyed strong growth to new heights driven by several factors, such as the COVID-19 pandemic or advances in technology, allowing consumers and patients to continuously record health data by themselves. This does not come without its challenges, however. A literature review was completed and highlights usability gaps when using wearables or home use medical devices in a virtual environment. Based on these findings, the Pi-CON methodology was applied to close these gaps by utilizing a novel sensor that allows the acquisition of vital signs at a distance, without any sensors touching the patient. Pi-CON stands for passive, continuous and non-contact, and describes the ability to acquire vital signs continuously and passively, with limited user interaction. The preference of vital sign acquisition with a newly developed sensor was tested and compared to vital sign tests taken with patient generated health-data devices (ear thermometer, pulse oximeter) measuring heart rate, respiratory rate and body temperature. In addition, the amount of operator errors and the user interfaces were tested and compared. Results show that participants preferred vital signs acquisition with the novel sensor and the developed user interface of the sensor. Results also revealed that participants had a mean error of .85 per vital sign measurement with the patient-generated health data devices and .33 with the developed sensor, confirming the beneficial impact available when using the developed sensor based on the Pi-CON methodology.

Estimating Stress in Online Meetings by Remote Physiological Signal and Behavioral Features

Work stress impacts people's daily lives. Their well-being can be improved if the stress is monitored and addressed in time. Attaching physiological sensors are used for such stress monitoring and analysis. Such approach is feasible only when the person is physically presented. Due to the transfer of the life from offline to online, caused by the COVID-19 pandemic, remote stress measurement is of high importance. This study investigated the feasibility of estimating participants' stress levels based on remote physiological signal features (rPPG) and behavioral features (facial expression and motion) obtained from facial videos recorded during online video meetings. Remote physiological signal features provided higher accuracy of stress estimation (78.75%) as compared to those based on motion (70.00%) and facial expression (73.75%) features. Moreover, the fusion of behavioral and remote physiological signal features increased the accuracy of stress estimation up to 82.50%. © 2022 Owner/Author.

A review of the effect of skin pigmentation on pulse oximeter accuracy.

Objective. Pulse oximetry is a non-invasive optical technique used to measure arterial oxygen saturation (SpO2) in a variety of clinical settings and scenarios. Despite being one the most significant technological advances in health monitoring over the last few decades, there have been reports on its various limitations. Recently due to the Covid-19 pandemic, questions about pulse oximeter technology and its accuracy when used in people with different skin pigmentation have resurfaced, and are to be addressed.Approach. This review presents an introduction to the technique of pulse oximetry including its basic principle of operation, technology, and limitations, with a more in depth focus on skin pigmentation. Relevant literature relating to the performance and accuracy of pulse oximeters in populations with different skin pigmentation are evaluated.Main Results. The majority of the evidence suggests that the accuracy of pulse oximetry differs in subjects of different skin pigmentations to a level that requires particular attention, with decreased accuracy in patients with dark skin.Significance. Some recommendations, both from the literature and contributions from the authors, suggest how future work could address these inaccuracies to potentially improve clinical outcomes. These include the objective quantification of skin pigmentation to replace currently used qualitative methods, and computational modelling for predicting calibration algorithms based on skin colour.

A real-time heart rate estimation framework based on a facial video while wearing a mask.

BACKGROUND: The imaging photoplethysmography (iPPG) method is a non-invasive, non-contact measurement method that uses a camera to detect physiological indicators. On the other hand, wearing a mask has become essential today when COVID-19 is rampant, which has become a new challenge for heart rate (HR) estimation from facial videos recorded by a camera. OBJECTIVE: The aim is to propose an iPPG-based method that can accurately estimate HR with or without a mask. METHODS: First, the facial regions of interest (ROI) were divided into two sub-ROIs, and the original signal was obtained through spatial averaging with different weights according to the result of judging whether wearing a mask or not, and the CDF, which emphasizes the main component signal, was combined with the improved POS suitable for real-time HR estimation to obtain the noise-removed BVP signal. RESULTS: For self-collected data while wearing a mask, MAE, RMSE, and ACC were 1.09 bpm, 1.44 bpm, and 99.08%, respectively. CONCLUSION: Experimental results show that the proposed framework can estimate HR stably in real-time in both cases of wearing a mask or not. This study expands the application range of HR estimation based on facial videos and has very practical value in real-time HR estimation in daily life.

A Frequency Estimation Scheme Based on Gaussian Average Filtering Decomposition and Hilbert Transform: With Estimation of Respiratory Rate as an Example.

Frequency estimation plays a critical role in vital sign monitoring. Methods based on Fourier transform and eigen-analysis are commonly adopted techniques for frequency estimation. Because of the nonstationary and time-varying characteristics of physiological processes, time-frequency analysis (TFA) is a feasible way to perform biomedical signal analysis. Among miscellaneous approaches, Hilbert-Huang transform (HHT) has been demonstrated to be a potential tool in biomedical applications. However, the problems of mode mixing, unnecessary redundant decomposition and boundary effect are the common deficits that occur during the procedure of empirical mode decomposition (EMD) or ensemble empirical mode decomposition (EEMD). The Gaussian average filtering decomposition (GAFD) technique has been shown to be appropriate in several biomedical scenarios and can be an alternative to EMD and EEMD. This research proposes the combination of GAFD and Hilbert transform that is termed the Hilbert-Gauss transform (HGT) to overcome the conventional drawbacks of HHT in TFA and frequency estimation. This new method is verified to be effective for the estimation of respiratory rate (RR) in finger photoplethysmography (PPG), wrist PPG and seismocardiogram (SCG). Compared with the ground truth values, the estimated RRs are evaluated to be of excellent reliability by intraclass correlation coefficient (ICC) and to be of high agreement by Bland-Altman analysis.

Camera-Based Blood Pressure Estimation via Windkessel Model and Waveform Features

Blood pressure (BP) is generally regarded as the vital sign most strongly correlated with human health. However, for decades, BP measurement has involved a cuff, which causes discomfort and even carries a risk of infection, given the current prevalence of COVID-19. Some studies address these problems using remote photoplethysmography (rPPG), which has shown great success in heart rate detection. Nevertheless, these approaches are not robust, and few have been evaluated with a sufficiently large dataset. We propose an rPPG-based BP estimation algorithm that predicts BP by leveraging the Windkessel model and hand-crafted waveform characteristics. A waveform processing procedure is presented for the rPPG signals to obtain a robust waveform template and thus extract BP-related features. Redundant and unstable features are eliminated via Monte Carlo simulation and according to their relationship with latent parameters (LSs) in the Windkessel model. For a comprehensive evaluation, the Chiao Tung BP (CTBP) dataset was constructed. The experiment was conducted over a four-week period of time to evaluate the validity period of the personalization in our system. On all the data, the proposed method outperforms the benchmark algorithms and yields mean absolute errors (MAEs) of 6.48 and 5.06 mmHg for systolic BP (SBP) and diastolic BP (DBP), respectively. The performance achieves a 'B' grade according to the validation protocol from the British Hypertension Society (BHS) for both SBP and DBP. © 1963-2012 IEEE.

DNN based reliability evaluation for telemedicine data.

Telemedicine data are measured directly by untrained patients, which may cause problems in data reliability. Many deep learning-based studies have been conducted to improve the quality of measurement data. However, they could not provide an accurate basis for judgment. Therefore, this study proposed a deep neural network filter-based reliability evaluation system that could present an accurate basis for judgment and verified its reliability by evaluating photoplethysmography signal and change in data quality according to judgment criteria through clinical trials. In the results, the deviation of 3% or more when the oxygen saturation was judged as normal according to each criterion was 0.3% and 0.82% for criteria 1 and 2, respectively, which was very low compared to the abnormal judgment (3.86%). The deviation of diastolic blood pressure (≥ 10 mmHg) according to criterion 3 was reduced by about 4% in the normal judgment compared to the abnormal. In addition, when multiple judgment conditions were satisfied, abnormal data were better discriminated than when only one criterion was satisfied. Therefore, the basis for judging abnormal data can be presented with the system proposed in this study, and the quality of telemedicine data can be improved according to the judgment result.

A Multisensor System Embedded in a Computer Mouse for Occupational Stress Detection.

Occupational stress is a major challenge in modern societies, related with many health and economic implications. Its automatic detection in an office environment can be a key factor toward effective management, especially in the post-COVID era of changing working norms. The aim of this study is the design, development and validation of a multisensor system embedded in a computer mouse for the detection of office work stress. An experiment is described where photoplethysmography (PPG) and galvanic skin response (GSR) signals of 32 subjects were obtained during the execution of stress-inducing tasks that sought to simulate the stressors present in a computer-based office environment. Kalman and moving average filters were used to process the signals and appropriately formulated algorithms were applied to extract the features of pulse rate and skin conductance. The results found that the stressful periods of the experiment significantly increased the participants' reported stress levels while negatively affecting their cognitive performance. Statistical analysis showed that, in most cases, there was a highly significant statistical difference in the physiological parameters measured during the different periods of the experiment, without and with the presence of stressors. These results indicate that the proposed device can be part of an unobtrusive system for monitoring and detecting the stress levels of office workers.

Cost-Effective Solution of Remote Photoplethysmography Capable of Real-Time, Multi-Subject Monitoring with Social Distancing.

Recent advances in remote-photoplethysmography (rPPG) have enabled the measurement of heart rate (HR), oxygen saturation (SpO2), and blood pressure (BP) in a fully contactless manner. These techniques are increasingly applied clinically given a desire to minimize exposure to individuals with infectious symptoms. However, accurate rPPG estimation often leads to heavy loading in computation that either limits its real-time capacity or results in a costly setup. Additionally, acquiring rPPG while maintaining protective distance would require high resolution cameras to ensure adequate pixels coverage for the region of interest, increasing computational burden. Here, we propose a cost-effective platform capable of the real-time, continuous, multi-subject monitoring while maintaining social distancing. The platform is composed of a centralized computing unit and multiple low-cost wireless cameras. We demonstrate that the central computing unit is able to simultaneously handle continuous rPPG monitoring of five subjects with social distancing without compromising the frame rate and rPPG accuracy.

Camera-based Blood Pressure Estimation via Windkessel Model and Waveform Features

Blood pressure (BP) is generally regarded as the vital sign most strongly correlated with human health. However, for decades, BP measurement has involved a cuff, which causes discomfort and even carries a risk of infection, given the current prevalence of COVID-19. Some studies address these problems using remote photoplethysmography (rPPG), which has shown great success in heart rate detection. Nevertheless, these approaches are not robust, and few have been evaluated with a sufficiently large dataset. We propose an rPPG-based BP estimation algorithm that predicts BP by leveraging the Windkessel model and hand-crafted waveform characteristics. A waveform processing procedure is presented for the rPPG signals to obtain a robust waveform template and thus extract BP-related features. Redundant and unstable features are eliminated via Monte Carlo simulation and according to their relationship with latent parameters in the Windkessel model. For a comprehensive evaluation, the Chiao Tung Blood Pressure (CTBP) dataset was constructed. The experiment was conducted over a four week period of time to evaluate the validity period of the personalization in our system. On all the data, the proposed method outperforms the benchmark algorithms and yields mean absolute errors of 6.48 mmHg and 5.06 mmHg for SBP and DBP, respectively. The performance achieves a “B”grade according to the validation protocol from the British Hypertension Society for both SBP and DBP. IEEE

Comparison and evaluation of IPPG methods for HR estimation under different face regions

Non-contact heart rate measurement reaches a higher level in scientific research;this field presents important advantages in our life, such as human-machine interaction, medical applications, especially in the current situation of the world suffering from COVID-19 pandemic. In recent years, several techniques of extracting the imaging photoplethysmography (ippg) signals from facial videos have been proposed and developed. Based on this, we performed a study and evaluation of the four most well known heart rate estimation methods such as Green, ICA, POS and CHROM in two accessible public datasets MAHNOB-HCI and UBFC-Phys under two different facial regions to enable researchers to develop and use them in real applications. The results show that the video imaging condition and the correct face region detection step play an important role in the accuracy of heart rate estimation. © 2022 IEEE.

Home Healthcare Technologies and Services: HeartRate Fetus Monitoring System Using an MCU ESP8266 Node

In this paper, a preliminary implementation of a system monitoring the fetus's heart rate (FHR) has been designed and implemented as a mobile wearable measuring system with remote sensing specifically developed on Node MCU ESP8266 (ESP). In particular, the proposed system uses sensors for heart rate, humidity, temperature, and a transceiver module. The transceiver module is capable of efficient data transmission to a remote server station using an IEEE 802.11 b/g/n protocol - based on the wireless network. A major benefit is that the patient's data is monitored at distance using an IoT device. Hence, it complies with the health safety distance measures required due to various situations, including that of the COVID-19 pandemic. The proposed implementation has been proven to be efficient in terms of hardware simplicity and cost-effectiveness and is accompanied by preliminary accurate measurements of the FHR. © 2022 IEEE.

ReViSe: Remote Vital Signs Measurement Using Smartphone Camera

Remote Photoplethysmography (rPPG) is a fast, effective, inexpensive and convenient method for collecting biometric data as it enables vital signs estimation using face videos. Remote contactless medical service provisioning has proven to be a dire necessity during the COVID-19 pandemic. We propose an end-to-end framework to measure people's vital signs including Heart Rate (HR), Heart Rate Variability (HRV), Oxygen Saturation (SpO2) and Blood Pressure (BP) based on the rPPG methodology from the video of a user's face captured with a smartphone camera. We extract face landmarks with a deep learning-based neural network model in real-time. Multiple face patches also called Regions-of-Interest (RoIs) are extracted by using the predicted face landmarks. Several filters are applied to reduce the noise from the RoIs in the extracted cardiac signals called Blood Volume Pulse (BVP) signal. The measurements of HR, HRV and SpO2 are validated on two public rPPG datasets namely the TokyoTech rPPG and the Pulse Rate Detection (PURE) datasets, on which our models achieved the following Mean Absolute Errors (MAE): a) for HR, 1.73Beats-Per-Minute (bpm) and 3.95bpm respectively;b) for HRV, 18.55ms and 25.03ms respectively, and c) for SpO2, an MAE of 1.64% on the PURE dataset. We validated our end-to-end rPPG framework, ReViSe, in daily living environment, and thereby created the Video-HR dataset. Our HR estimation model achieved an MAE of 2.49bpm on this dataset. Since no publicly available rPPG datasets existed for BP measurement with face videos, we used a dataset with signals from fingertip sensor to train our deep learning-based BP estimation model and also created our own video dataset, Video-BP. On our Video-BP dataset, our BP estimation model achieved an MAE of 6.7mmHg for Systolic Blood Pressure (SBP), and an MAE of 9.6mmHg for Diastolic Blood Pressure (DBP). ReViSe framework has been validated on datasets with videos recorded in daily living environment as opposed to less noisy laboratory environment as reported by most state-of-the-art techniques.

Evaluation of a Camera-Based Monitoring Solution Against Regulated Medical Devices to Measure Heart Rate, Respiratory Rate, Oxygen Saturation, and Blood Pressure.

Background Regularly monitoring common physiological signs, including heart rate, blood pressure, and oxygen saturation, can effectively prevent or detect several potential conditions. In particular, cardiovascular diseases (CVDs) are a worldwide concern. According to the World Health Organization, 31% of all deaths worldwide are from CVDs. Recently, the coronavirus disease 2019 pandemic has increased the interest in remote monitoring. At present, contact devices are required to extract most of an individual's physiological information, which can be inconvenient for users and may cause discomfort. Methodology However, remote photoplethysmography (rPPG) technology offers a solution for this issue, enabling contactless monitoring of the blood volume pulse signal using a regular camera. Ultimately, it can provide the same physiological information as a contact device. In this paper, we propose an evaluation of Vastmindz's rPPG technology against medical devices in a clinical environment with a variety of subjects in a wide range of age, height, weight, and baseline vital signs. Results This study confirmed the findings that the contactless technology for the estimation of vitals proposed by Vastmindz was able to estimate heart rate, respiratory rate, and oxygen saturation with a mean error of ±3 units as well as ±10 mmHg for systolic and diastolic blood pressure. Conclusions Reported results have shown that Vastmindz's rPPG technology was able to meet the initial hypothesis and is acceptable for users who want to understand their general health and wellness.

Measurement of Vital Signs by Lifelight Software in Comparison to Standard of Care Multisite Development (VISION-MD): Protocol for an Observational Study.

BACKGROUND: Measuring vital signs (VS) is an important aspect of clinical care but is time-consuming and requires multiple pieces of equipment and trained staff. Interest in the contactless measurement of VS has grown since the COVID-19 pandemic, including in nonclinical situations. Lifelight is an app being developed as a medical device for the contactless measurement of VS using remote photoplethysmography (rPPG) via the camera on smart devices. The VISION-D (Measurement of Vital Signs by Lifelight Software in Comparison to the Standard of Care-Development) and VISION-V (Validation) studies demonstrated the accuracy of Lifelight compared with standard-of-care measurement of blood pressure, pulse rate, and respiratory rate, supporting the certification of Lifelight as a class I Conformité Européenne (CE) medical device. OBJECTIVE: To support further development of the Lifelight app, the observational VISION Multisite Development (VISION-MD) study is collecting high-quality data from a broad range of patients, including those with VS measurements outside the normal healthy range and patients who are critically ill. METHODS: The study is recruiting adults (aged ≥16 years) who are inpatients (some critically ill), outpatients, and healthy volunteers, aiming to cover a broad range of normal and clinically relevant VS values; there are no exclusion criteria. High-resolution 60-second videos of the face are recorded by the Lifelight app while simultaneously measuring VS using standard-of-care methods (automated sphygmomanometer for blood pressure; finger clip sensor for pulse rate and oxygen saturation; manual counting of respiratory rate). Feedback from patients and nurses who use Lifelight is collected via a questionnaire. Data to estimate the cost-effectiveness of Lifelight compared with standard-of-care VS measurement are also being collected. A new method for rPPG signal processing is currently being developed, based on the identification of small areas of high-quality signals in each individual. Anticipated recruitment is 1950 participants, with the expectation that data from approximately 1700 will be used for software development. Data from 250 participants will be retained to test the performance of Lifelight against predefined performance targets. RESULTS: Recruitment began in May 2021 but was hindered by the restrictions instigated during the COVID-19 pandemic. The development of data processing methodology is in progress. The data for analysis will become available from September 2022, and the algorithms will be refined continuously to improve clinical accuracy. The performance of Lifelight compared with that of the standard-of-care measurement of VS will then be tested. Recruitment will resume if further data are required. The analyses are expected to be completed in early 2023. CONCLUSIONS: This study will support the refinement of data collection and processing toward the development of a robust app that is suitable for routine clinical use. TRIAL REGISTRATION: ClinicalTrials.gov NCT04763746; https://clinicaltrials.gov/ct2/show/NCT04763746. INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID): DERR1-10.2196/41533.

Measurement of Blood Oxygen based on Remote-Photoplethysmography

SpO2, also known as blood oxygen saturation, is a vital physiological indicator in clinical care. Since the outbreak of COVID-19, silent hypoxia has been one of the most serious symptoms. This symptom makes the patient's SpO2 drop to an extremely low level without discomfort and causes medical care delay for many patients. Therefore, regularly checking our SpO2 has become a very important matter. Recent work has been looking for convenient and contact-free ways to measure SpO2 with cameras. However, most previous studies were not robust enough and didn't evaluate their algorithms on the data with a wide SpO2 range. In this paper, we proposed a novel non-contact method to measure SpO2 by using the weighted K-nearest neighbors (KNN) algorithm. Five features extracted from the RGB traces, POS, and CHROM signals were used in the KNN model. Two datasets using different ways to lower the SpO2 were constructed for evaluating the performance. The first one was collected through the breath-holding experiment, which induces more motion noise and confuses the actual blood oxygen features. The second dataset was collected at Song Syue Lodge, which locates at an elevation of 3150 meters and has lower oxygen concentration in the atmosphere making the SpO2 drop between the range of 80% to 90% without the need of holding breath. The proposed method outperforms the benchmark algorithms on the leave-one-subject-out and cross-dataset validation. © 2022 IEEE.

Augmentation of rPPG Benchmark Datasets: Learning to Remove and Embed rPPG Signals via Double Cycle Consistent Learning from Unpaired Facial Videos

Remote estimation of human physiological condition has attracted urgent attention during the pandemic of COVID-19. In this paper, we focus on the estimation of remote photoplethysmography (rPPG) from facial videos and address the deficiency issues of large-scale benchmarking datasets. We propose an end-to-end RErPPG-Net, including a Removal-Net and an Embedding-Net, to augment existing rPPG benchmark datasets. In the proposed augmentation scenario, the Removal-Net will first erase any inherent rPPG signals in the input video and then the Embedding-Net will embed another PPG signal into the video to generate an augmented video carrying the specified PPG signal. To train the model from unpaired videos, we propose a novel double-cycle consistent constraint to enforce the RErPPG-Net to learn to robustly and accurately remove and embed the delicate rPPG signals. The new benchmark “Aug-rPPG dataset” is augmented from UBFC-rPPG and PURE datasets and includes 5776 videos from 42 subjects with 76 different rPPG signals. Our experimental results show that existing rPPG estimators indeed benefit from the augmented dataset and achieve significant improvement when fine-tuned on the new benchmark. The code and dataset are available at https://github.com/nthumplab/RErPPGNet. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Design and Implementation of an Interactive Photoplethysmography and Galvanic Skin Response Based Gamepad

Mental disorders in the young adult population are becoming more frequent, largely due to the COVID-19 pandemic. This has led to the need to find new ways to adapt to therapeutic methods, offering greater attractiveness for this age range, and in many studies, it has been reported that this can be achieved thanks to video games. In this work, a controller design for video games that allows to obtain some of the most relevant biological signals of the relationship between the physiological state and the mental state of the user is proposed. An accessible and non-invasive instrument was built, in the form of a video game controller, to make measurements of heart rate and the galvanic response of the skin, two physiological variables that play a vital role in determining a person’s emotional state, that allows, in turn, to play video games that are designed to be able to perform actions based on the measurements of biosignals, such as modifying the difficulty, improving the user experience, etc. Making use of two biosignal sensors (photoplethysmography and galvanic skin response), the controller is developed to offer non-invasive biofeedback while playing computer video games, which provides an effective approach to developing interactive and customizable diagnostic and therapeutic psychological tools. This work, which involves the unification of various ideas and fields, could mean an advance in the field of the development of digital alternatives for therapies related to mental health, as well as a tool that allows a greater approach on the part of the community to which it is focused. This may mean that, in future developments, there is greater cohesion and a greater boom in treatments for people considered young adults. © 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.

Contactless blood oxygen estimation from face videos: A multi-model fusion method based on deep learning.

Blood Oxygen ( SpO 2 ), a key indicator of respiratory function, has received increasing attention during the COVID-19 pandemic. Clinical results show that patients with COVID-19 likely have distinct lower SpO 2 before the onset of significant symptoms. Aiming at the shortcomings of current methods for monitoring SpO 2 by face videos, this paper proposes a novel multi-model fusion method based on deep learning for SpO 2 estimation. The method includes the feature extraction network named Residuals and Coordinate Attention (RCA) and the multi-model fusion SpO 2 estimation module. The RCA network uses the residual block cascade and coordinate attention mechanism to focus on the correlation between feature channels and the location information of feature space. The multi-model fusion module includes the Color Channel Model (CCM) and the Network-Based Model(NBM). To fully use the color feature information in face videos, an image generator is constructed in the CCM to calculate SpO 2 by reconstructing the red and blue channel signals. Besides, to reduce the disturbance of other physiological signals, a novel two-part loss function is designed in the NBM. Given the complementarity of the features and models that CCM and NBM focus on, a Multi-Model Fusion Model(MMFM) is constructed. The experimental results on the PURE and VIPL-HR datasets show that three models meet the clinical requirement(the mean absolute error ⩽ 2%) and demonstrate that the multi-model fusion can fully exploit the SpO 2 features of face videos and improve the SpO 2 estimation performance. Our research achievements will facilitate applications in remote medicine and home health.